Concerning the problems of the low accuracy and high requirements for data in the existing building identification and extraction methods from high-resolution remotely sensed imagery, a new method based on Neighborhood Total Variation (NTV) and Potential Histogram Function (PHF) was proposed. Firstly, the value of weighted NTV likelihood function for each pixel of a remotely sensed imagery was calculated, the segmentation was done with region growing method, and the candidate buildings were selected from the segmentation results with the constraints of rectangular degree and aspect ratio. Then, the shadows were detected automatically. At last, shadows were processed with morphology operations. The buildings were extracted by computing the adjacency relationship of the processed shadows and candidate buildings, and the building boundaries were fitted with the minimum enclosing rectangle. For verifying the validity of the proposed method, nine representative sub-images were chosen from PLEIADES images covering Shenzhen for experiment. The experimental results show that, the average precision and recall of the proposed method are 97.71% and 84.21% for the object-based evaluation, and the proposed method has increased the overall performance F ₁by more than 10% compared with two other building extraction methods based on level set and color invariant feature.